Skip to content

Cache choice weight tables, and stop wasting SQL - #174

Open
yhong123 wants to merge 1 commit into
mainfrom
yhong123/147-optimise_choice_proposers
Open

yhong123 wants to merge 1 commit into
mainfrom
yhong123/147-optimise_choice_proposers

Conversation

@yhong123

Copy link
Copy Markdown
Collaborator

Summary

Closes #147.
Works towards #160.

weighted_choice/zipf_choice_direct were rebuilding their weight tables on every single generated row (#147). Also, there is a separate, pre-existing bottleneck in ChoiceProposerFactory.get_proposers — the SQL step that builds proposers in the first place — dominates total time for high-cardinality columns and was masking the improvement. This PR bundles all three fixes together.

Changes

1. weighted_choice/zipf_choice_direct per-row caching

datafaker/providers.py

  • weighted_choice(a) previously rebuilt its (values, counts) lists from scratch on every call, and random.choices(..., weights=...) redid its own O(n) cumulative-weight setup every time too, even though a is the same list object across an entire generate_data(count) loop.
  • Added an id(a)-keyed cache (DistributionProvider._weighted_choice_cache) so the derived (values, cumulative_counts) are only rebuilt when a is actually a new list, and switched random.choices to cum_weights= (precomputed) instead of weights= (recomputed every call).
  • Same idea for zipf_choice_direct: zipf_cumulative_weights(size) is functools.cached by size.

Measured (generate_data(4000), concept_id column, isolated from any SQL changes):

proposer before after speedup
zipf_choice [sampled] 0.0483s 0.0043s 11x
weighted_choice [sampled] 0.0983s 0.0054s 18x
choice [sampled] 0.0008s 0.0018s unaffected (never the bottleneck)

2. Stop sorting stmt_count results that get thrown away

datafaker/proposers/choice.py

  • ChoiceProposerFactory.get_proposers's exact stmt_count query had ORDER BY count DESC, sorting the entire table in the database — even though, for any column over MAXIMUM_CHOICES (500) distinct values (the common case for a real column), the result is discarded unseen once the row-count cap check fails.
  • Removed the SQL-side ORDER BY; when the cap check does pass (≤500 rows), the small result is sorted in Python instead, preserving ZipfChoiceProposer's most-frequent-first ordering requirement exactly.

Measured (get_proposers, concept_id, a near-unique PK column): ~1.0s → 0.397s.

3. Skip the exact query entirely for high-cardinality columns

datafaker/proposers/choice.py

  • For an unindexed, very-high-cardinality column (e.g. concept_name), stmt_count's full-table GROUP BY was taking 12.6s, confirmed via EXPLAIN (ANALYZE, BUFFERS). Its result is discarded unseen anyway, same as above.
  • get_proposers now runs the cheap, bounded SAMPLE_COUNT-row sample query first. If that sample comes back ≥90% distinct (EXACT_SKIP_SAMPLE_DISTINCT_RATIO), the exact query is skipped outright, both unaffordable and pointless in that case.
  • The 90% threshold is deliberately conservative, so this only fires when cardinality is genuinely far over the cap (MAXIMUM_CHOICES).

Measured (get_proposers):

column before (Fix 1 only) after (+ skip heuristic) notes
concept_name (high cardinality, unindexed) 12.654s 0.972s ~13x speedup
concept_id (near-unique PK) 0.397s 0.455s no real regression — its exact query was already cheap via PK-index early exit, so skipping it saves little; difference is DB timing noise

Testing

  • New regression tests in tests/test_proposer.py:
    • ChoiceProposerFactoryOrderingTests — values still come back most-frequent-first after removing the SQL-side sort (verified to fail without the fix).
    • ChoiceProposerFactoryExactSkipTests — the exact query is skipped for a high-cardinality column and still runs for a low-cardinality one (captured via SQLAlchemy's before_cursor_execute; verified to fail without the fix).
  • Full test suite passes.
  • All numbers above were measured against a live OMOP-mimic Postgres database, not just synthetic test fixtures.

@yhong123 yhong123 self-assigned this Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dist_gen.weighted_choice/zipf_choice rebuild their weight tables on every single call

1 participant